Emergence of Selective Invariance in Hierarchical Feed Forward Networks

Many theories have emerged which investigate how invariance is generated in hierarchical networks through simple schemes such as max and mean pooling. The restriction to max/mean pooling in theoretical and empirical studies has diverted attention away from a more general way of generating invariance to nuisance transformations. In this exploratory study, we study the conjecture that hierarchically building selective invariance is important for pattern recognition. We define selective invariance as carefully choosing the range of the transformation to be invariant to at each layer of a hierarchical network. For the purpose of our study, we utilize a novel method called adaptive pooling where the pooling weights are not constrained and in fact can adapt their pooling regions to the data. These networks with the adapted pooling regions maintain performances on object categorization tasks comparable to max/mean pooling networks despite being more prone to overfitting. Interestingly, adaptive pooling regions can converge to mean pooling (even when initialized with random pooling regions), find more general linear pooling schemes or even decide not to pool at all. The pooling regions that emerge from the data are not random but rather contiguous, illustrating invariance to contiguous ranges of transformations. We illustrate the general notion of selective invariance through object categorization experiments on largescale datasets such as SVHN and ILSVRC 2012.